Interactive tandem networks and the sequential learning problem

This paper presents a novel connectionist architecture to handle the "sensitivity-stability" problem and, in particular, an extreme manifestation of the problem, catastrophic interference. This architecture, called an interactive tandem-network (ITN) architecture, consists of two continually interacting networks, one — the LTM network — dynamically storing "prototypes" of the patterns learned, the other — the STM network — being responsible for "short-term" learning of new patterns. Prototypes stored in the LTM network influence hidden-layer representations in the STM network and, conversely, newly learned representations in the STM network gradually modify the more stable LTM prototypes. As prototypes are learned by the LTM network, they are dynamically constrained to maximize mutual orthogonality. This system of tandem networks performs particularly well on the problem of catastrophic interference. It also produces "long-term" representations that are stable in the face of new input and "short-term" representations that remain sensitive to new input. Justification for this type of architecture is similar to that given recently by McClelland, McNaughton, & O'Reilly (1994) in arguing for the necessary complementarity of the hippocampal (short-term memory) and neocortical (longterm memory) systems.